Update doc with DuckDB version requirement for querying JSON columns in the Tailpipe database #61
Conversation
Preview Available 🚀Commit Author: Ved misra Preview Link: tailpipe-io-git-docs-update-duckdb-version-requirement-turbot.vercel.app |
Co-authored-by: Puskar Basu <45908484+pskrbasu@users.noreply.github.com>
| @@ -36,9 +36,12 @@ tailpipe connect --from 2025-01-01 | |||
| ``` | |||
|
|
|||
| > [!NOTE] | |||
| > You could use this connection string with DuckDB: | |||
| > You can use this connection string with DuckDB to directly query the Tailpipe database. | |||
| To ensure compatibility with tables that include JSON columns, make sure you’re using DuckDB version 1.1.3 or later. | |||
There was a problem hiding this comment.
Is that because the JSON extension is installed/enabled by default? Could you also install it in an older version, or disable it in a newer version? Are there any other extensions that are required/recommended?
There was a problem hiding this comment.
Hi @johnsmyth, sorry for the delayed response. After digging deeper, we realized that the behavior mentioned in this issue comment is not directly related to the JSON column itself.
We created a table in a mydb.duckdb file with a JSON column, and everything works fine there. DuckDB v1.0.0 supports JSON and comes with a built-in, preloaded JSON extension, as documented here. The same extension exists in DuckDB v1.3.2 as well (reference), and both versions behave consistently.
The actual issue is that aws_cloudtrail_log is a view over parquet files defined by DuckDB v1.3.2 in Tailpipe. When we attempt to query this view using DuckDB v1.0.0, the JSON fields are downgraded during binding (e.g., JSON → VARCHAR), which leads to a Binder error during query execution. However, dropping the existing view and redefining the VARCHAR fields as JSON resolves the issue. Hence we require duckdb v1.1.3 and above.
Issue Link: turbot/tailpipe#164